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F.XHCUTIVE  SUMMARY 


Purpose 

The  aim  of  this  project  was  to  assess  the  quality  of  ARI’s  current  survey  programs,  make 
recommendations  for  improving  them,  and  to  draft  regulations  that  incorporated  these 
recommendations  and  brought  the  regulations  up  to  date. 


Methods 

We  gathered  information  about  ARI’s  current  altitudinal,  command  climate,  and 
occupational  analysis  studies  by  examining  survey  documentation  and  speaking  with  the  stall 
who  carry  out  the  studies.  We  also  gathered  similar  information  about  a  number  of  comparable 
surveys  done  by  the  other  services,  academic  survey  organizations,  and  private  firms.  In 
addition,  we  identified  the  users  of  the  ARI  data  and  conducted  a  focus  group  and  a  small 

telephone  study  to  assess  their  satisfaction  with  ARI’s  surveys. 


Findings 

We  found  that  ARI  used  sound  methods  comparable  to  those  used  by  other  survey 
organizations  and  it  achieved  similar  response  rates.  ARI’s  customers  expressed  a  high  level  of 
satisfaction  both  in  the  focus  group  discussion  and  the  more  formal  customer  satisfaction  survey. 
There  were  several  areas  where  we  recommended  improvements,  including  steps  intended  to 
improve  ARI’s  questionnaires  and  the  response  rates  to  its  surveys  and  other  steps  intended  to 
increase  its  use  of  automation.  In  many  cases,  our  recommendations  supported  initiatives  ARI 

had  already  begun. 


Conclusions 

We  presented  a  number  of  recommendations  to  ARI.  These  included  suggestions  for 
improving  its  survey  response  rates,  standardizing  its  pretesting  procedures,  adopting  new  data 
collection  technologies,  creating  a  quick  turnaround  capability,  standardizing  deliverables,  and 
obtaining  systematic  customer  feedback. 
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INTRODUCTION 


In  September  of  1998,  The  Gallup  Organization  was  awarded  a  Task  Order  Contract  under 
a  subcontract  with  Personnel  Decisions  Research  Institutes  (PDRI)  to  carry  out  an  assessment  of 
the  Army  Research  Institute’s  (ARI)  Behavioral  and  Social  Science  survey  research  program. 

This  review  had  three  major  purposes.  The  first  was  to  evaluate  ARI  s  surveys  by  comparing 
them  to  industry  benchmarks.  The  second  was  to  provide  the  basis  for  recommendations  about 
improving  ARI’s  current  practices.  The  final  objective  was  to  update  the  regulations  under 
which  ARI  operates,  particularly  the  basic  regulation  (AR  600-46)  governing  ARI’s  surveys;  this 
regulation  has  not  been  overhauled  in  more  than  20  years. 

The  original  Task  Order  Request  for  the  project  called  for  six  subtasks: 

1.  A  review  of  ARI’s  current  policies  and  practices  for  conducting  its  attitude  and 
opinion  and  occupational  surveys; 

2.  The  identification  of  “best  practices”  used  in  similar  surveys  done  by  other 
governmental  and  private  survey  research  organizations; 

3.  An  assessment  of  the  level  of  satisfaction  of  ARI’s  current  customers  with  ARI’s 
surveys  and  the  reports  and  other  products  based  on  them; 

4.  A  briefing  of  the  Army’s  Attitude  and  Opinion  Survey  Advisory  Group; 

5.  Preoaration  of  a  report  based  on  the  findings  from  the  first  three  Tasks,  and 

6.  Development  of  draft  regulations  to  update  the  existing  regulations,  incorporate  any 
recommended  changes,  and  resolve  other  outstanding  issues  not  addressed  in  the 
current  regulations. 

Gallup  staff  gathered  detailed  information  about  seven  ARI  surveys  for  the  first  of  these 
tasks,  and  gathered  information  on  11  attitude,  employee  morale,  and  occupational  analysis 
surveys  done  outside  of  ARI  as  part  of  the  second  subtask.  To  assess  customer  satisfaction,  we 
held  a  focus  group  with  five  ARI  customers  and  interviewed  31  ARI  customers  by  telephone 
using  a  questionnaire  modeled  on  the  one  Gallup  uses  to  assess  the  satisfaction  of  its  own 
customers.  On  March  9, 1999  we  presented  our  findings  to  the  Survey  Advisory  Group  during  a 
briefing  at  the  Pentagon.  This  report  fulfills  the  requirement  in  subtask  5  and  includes  a  draft  of 
the  revised  regulations. 

The  balance  of  this  report  presents  a  more  detailed  description  of  each  of  the  first  three  of 
these  activities.  It  concludes  with  a  list  of  our  recommendations  and  a  discussion  of  the  draft 
regulation  we  are  proposing. 
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ART’S  CURRENT  PRACTICES 


Our  first  task  was  to  determine  the  current  practices  of  ARI’s  attitude  and  opinion  survey 
programs.  To  accomplish  this,  we  developed  a  conceptual  framework  that  broke  the  process  of 
conducting  a  survey  into  several  distinct  steps  (from  frame  development  to  reporting),  each  with 
its  own  challenges  and  potential  problems.  Our  plan  was  to  gather  information  on  how  each  step 
has  been  conducted  on  several  major  survey  studies  done  at  ARI.  Accordingly,  we  identified 
several  key  ARI  surveys  and  the  researchers  responsible  for  conducting  them.  We  paid 
particular  attention  to  any  apparent  challenges  or  difficulties  in  implementing  the  research  steps. 


Stases  in  Survey  Development 

For  each  study,  we  asked  the  lead  researcher  a  series  of  general  questions  about 
organizational  and  administrative  issues,  such  as  the  name  of  the  study,  the  general  purpose  of 
the  research,  the  study’s  customers,  and  the  regulations  and  clearance  procedures  for  the 
research.  Then,  we  asked  more  detailed  questions  about  the  main  steps  in  the  survey  research 
process.  Each  of  these  steps  can  affect  the  accuracy  of  the  survey  results.  Our  questions  sought 
to  uncover  the  methods  used  and  the  challenges  faced  in  each  step  of  the  process — frame 
development,  questionnaire  design,  sample  design,  data  collection,  data  processing,  and 
reporting. 


1)  Frame  development: 

Imnact  on  Survey  Error:  A  frame  is  an  implicit  or  explicit  list  of  the  population  from 
which  the  sample  for  a  particular  survey  is  drawn.  Coverage  errors  arise  if  some  portion  of 
the  population  of  interest  is  not  included  on  the  frame.  Because  of  their  omission  from  the 
frame,  these  members  of  the  population  can  never  be  selected  for  the  sample.  If  the 
information  on  the  frame  used  for  a  survey  is  not  kept  up  to  date,  these  inaccuracies  can 
also  produce  errors.  For  example,  if  the  locating  information  on  the  frame  is  out  of  date, 
nonresponse  can  result. 


Questions  Asked:  What  is  the  population  of  interest?  What  is  the  frame?  Are  there  any 
problems  with  the  frame,  such  as  poor  locating  information? 

2)  Questionnaire  design: 

Impact  on  Reporting  Errors:  Careful  design  of  a  survey  instrument  can  reduce 
measurement  error.  Research  has  shown  that  small,  apparently  innocuous  changes  in 
wording  of  a  question  can  lead  to  large  changes  in  responses  (Schuman  &  Presser,  1981; 
Sudman,  Bradburn,  &  Schwarz,  1996).  Cognitive  interviews,  focus  groups,  pre-testing, 
and/or  expert  review  panels  are  all  methods  used  to  improve  questionnaire  design. 
Minimizing  the  burden  or  length  of  the  survey  instrument  can  also  reduce  survey  error,  as 
a  long  survey  can  discourage  respondents  from  filling  it  out.  The  study  design  also  affects 
reporting  errors;  for  example,  longitudinal  surveys  are  prone  to  time-in-sample  effects. 
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which  can  affect  the  accuracy  of  the  answers. 

Questions  Asked:  What  types  of  pretesting  methods  are  used?  Are  the  reliability  and 
validity  of  items  tested?  What  is  the  length  or  burden  of  the  instrument?  What  is  the  study 
design  (cross-sectional,  longitudinal,  etc.)?  Are  there  any  rules  about  the  use  of  open- 
ended  items?  Are  standardized  scales  used?  Are  any  questions  from  previous  surveys 

used? 

3)  Sample  design: 

Impact  on  Survey  Error:  Several  features  of  the  sample  design  affect  the  precision  of  the 
estimates  derived  from  the  survey  data.  These  include  the  overall  sample  size,  the  use  of 
stratification  (that  is,  the  selection  of  separate  samples  within  different  subgroups  of  the 
population),  the  presence  of  clustering  in  the  data,  and  the  need  to  weight  the  data. 

Questions  Asked:  What  is  the  sample  size?  Are  weights  used?  Is  there  any  stratification? 
Clustering? 

4)  Data  collection: 

Impact  on  Survey  Error:  Data  collection  procedures  and  operations  have  a  direct  impact 
on  survey  response  rates.  Factors  such  as  the  mode  of  contact,  the  technology  used  to 
collect  the  data,  follow-up  strategies,  and  response  incentives  all  have  effects  on 
nonresponse.  For  instance,  respondents  tend  to  report  more  socially  undesirable  behavior 
in  a  self-administered  questionnaire  than  in  an  interviewer-administered  setting 
(Tourangeau  &  Smith,  1 996).  The  length  of  the  data  collection  period  can  also  effect 
nonresponse— short  data  collection  periods  may  systematically  exclude  those  who  are 
unavailable  during  the  survey  period.  Closely  related  to  this  is  the  extent  of  follow  up 
efforts.  Research  has  shown  that  multiple  mailings  of  a  questionnaire  can  increase 
response  rates  significantly  (Heberlein  &  Baumgartner,  1978). 

Questions  Asked:  What  was  the  mode  of  data  collection  (self-administered/telephone/in¬ 
person,  etc.)?  What  was  the  length  of  the  data  collection  period?  Was  any  form  of 
prenotification  used?  What  were  follow-up  efforts  were  made?  What  was  the  expected 
response  rate?  What  was  the  actual  response  rate? 


5)  Data  processing: 

Impact  on  Survey  Error:  Editing,  cleaning,  weighting  and  imputation  are  data  quality 
improvement  tools  by  which  erroneous,  highly  suspect,  or  missing  data  are  found  and,  if 
necessary,  corrected.  Such  data  processing  steps  provide  more  detailed  information  about 
the  quality  of  the  survey,  provide  basic  data  for  the  improvement  of  the  survey,  and  tidy  up 
the  data  so  further  processing  and  analysis  can  be  done.  Editing  should  not  change  survey 
results  in  an  unpredictable  manner. 
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Questions  Asked:  What  is  the  format  of  the  data  file?  What  cleaning,  editing,  imputation, 
and  weighting  methods  are  used,  if  any?  Are  open-ended  items  coded? 


6)  Reporting: 

Tmnact  on  Survey  Error:  The  final  product  of  most  surveys  is  a  set  of  statistics  derived 
from  the  data  at  hand,  often  summarized  as  a  report.  Generally,  the  reporting  process  does 
not  introduce  new  errors  into  the  survey  results,  but  should  (to  the  extent  possible) 
document  the  impact  of  errors  from  earlier  steps  in  the  process  of  conducting  the  survey. 

Questions  Asked:  What  kinds  of  reports  are  generated?  What  kinds  of  statistical  analyses 
are  performed? 


Findings 

The  typical  ARI  survey  is  a  self-administered  questionnaire,  usually  done  on  paper.  ARI  is 
increasingly  exploring  methods  of  electronic  data  collection  on  surveys  such  as  the  Command 
Climate  Survey  and  Occupational  Analysis.  Currently,  ARI  does  not  conduct  any  research  with 
the  help  of  an  interviewer,  such  as  telephone  interviews  or  in-person  surveys.  Instead,  surveys 
are  either  mailed  directly  to  the  respondent,  or  distributed  through  an  installation  or  commander. 
Average  response  rates  for  ARI  are  about  50%,  which  is  more  or  less  consistent  with  the  rates  we 
observed  in  our  survey  of  the  industry,  particularly  on  mail  surveys  (see  also  Brick,  Cantor, 
Kerwin,  Giambo,  and  Singer,  1997). 

We  will  now  describe  the  findings  from  interviews  with  the  lead  researchers  based  on  these 
six  areas  of  survey  research.  We  grouped  the  seven  ARI  surveys  we  examined  into  three 
categories:  1)  continuing  surveys,  which  included  attitude  surveys  done  on  a  regular  basis;  2) 
one-time  attitude  surveys,  which  are  special  surveys  carried  out  as  issues  arise;  and  3) 
occupational  analysis  studies,  which  are  special  class  of  surveys  conducted  by  ARI  to  gather 
information  about  the  specific  tasks  associated  with  particular  job  categories. 

Continuing  Surveys 

Gallup  examined  three  continuing  surveys — that  is,  surveys  done  by  ARI  on  a  fairly 
regular  basis.  These  include  the  Sample  Survey  of  Military  Personnel,  the  Survey  of  Officer 
Careers,  and  the  Command  Climate  Survey. 

The  Sample  Survey  of  Military  Personnel  (SSMP)  is  an  Army-wide  survey  conducted 
twice  a  year.  It  usually  addresses  about  10  to  20  topics  that  are  important  to  the  Army,  soldiers, 
and  their  families  and  collects  trend  data  on  a  number  of  issues.  It  takes  about  40  to  45  minutes 
to  complete  the  questionnaire,  which  is  usually  about  20  pages  long.  Most  of  the  questions  are 
not  new,  but  new  items  that  are  included  do  go  through  pretesting  to  make  sure  they  are  written 
properly.  The  sample  for  each  SSMP  is  stratified  by  rank  and  geographic  location.  Each  sample 
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covers  the  entire  Army  and  includes  about  10%  of  all  officers,  and  2-3%  of  enlisted  personnel. 
Paper,  scannable  questionnaires  are  sent  out  to  the  installations,  which  select  the  samples 
themselves  (using  the  last  digit  of  the  soldiers’  Social  Security  numbers).  ARI  does  not  track 
who  has  been  sampled  and  has  no  direct  contact  with  respondents.  As  a  result,  replacement 
mailings  are  not  sent.  For  soldiers  in  Europe  and  Korea,  ARI  draws  the  sample  based  on  the 
Army  Officer  and  Enlisted  Master  Files  and  mails  questionnaires  and  reminder  postcards  directly 
to  the  sample  members.  This  file  is  not  always  kept  up  to  date,  however,  and  some 
questionnaires  never  reach  the  intended  destination.  The  turnaround  time  on  this  survey  is 
usually  about  12-13  months.  Installations  bundle  surveys  to  mail  back  to  ARI,  which  can  slow 
down  returns.  Response  rates  are  generally  around  50%,  but  they  are  difficult  to  measure 
because  ARI  never  knows  exactly  how  many  surveys  were  mailed.  The  data  are  weighted  up  to 
Army  strength  for  each  individual  rank,  location,  and  gender.  Short  reports  are  generated  for 
each  topic  area,  and  the  relevant  data  may  be  sent  to  each  sponsor. 

The  Survey  on  Officer  Careers  (SOC)  provides  data  on  the  values,  attitudes,  and  career 
experiences  of  Army  officers.  The  data  are  used  to  understand  what  influences  the  career 
decisions  of  officers.  The  survey  includes  a  cross-sectional  sample  (that  is  sent  a  questionnaire 
just  once)  as  well  as  a  longitudinal  sample  that  provides  data  on  the  Army  experience  over  time. 
The  SOC  questionnaire  takes  about  40  to  45  minutes  to  fill  out.  It  is  mailed  to  sample  members; 
in  addition,  a  follow-up  postcard  is  mailed  approximately  two  weeks  after  the  questionnaire. 
Most  of  the  questions  come  from  the  longitudinal  survey,  but  there  are  usually  about  3  pages  of 
new  items  that  are  pretested;  focus  groups  are  being  done  now  to  help  develop  the  SOC  2000 
questionnaire.  The  longitudinal  sample  suffers  from  attrition  and  now  includes  about  3,400 
officers.  The  cross-sectional  sample  includes  7,500.  The  cross-sectional  sample  includes  all 
African-American  officers  as  well  as  all  white  female  officers;  the  remainder  of  the  cross- 
sectional  sample  stratified  by  rank,  race,  source  of  commission,  and  gender.  The  scannable 
questionnaires  are  mailed  out  directly  to  the  officers  using  the  Officer  Master  File,  but  this  file  is 
not  always  updated  with  home  addresses.  Response  rates  are  generally  around  50%. 

The  Command  Climate  Survey  helps  commanders  monitor  the  morale  of  their  units 
annually.  All  members  of  the  unit  are  asked  to  fill  out  the  survey.  It  takes  about  6  to  10  minutes 
to  complete  the  questionnaire.  This  is  a  completely  automated  survey  from  questionnaire  design 
to  administration  to  analysis  and  is  administered  at  the  unit  level  without  any  direct  involvement 
from  ARI.  The  survey  can  also  be  done  on  paper  if  the  unit  is  not  automated.  The  commanders 
have  the  opportunity  to  add  their  own  questions  to  the  questionnaire;  however,  there  is  no 
oversight  to  make  sure  these  questions  have  been  written  in  a  unbiased  way.  Commanders  can 
choose  from  seven  Likert-type  scales,  and  they  can  write  up  to  seven  closed-ended  and  three 
open-ended  questions.  (Closed-ended  questions  offer  explicit  response  categories;  open-ended 
questions  allow  respondents  to  generate  their  own  answer.)  These  surveys  are  a  census  of  all 
soldiers  in  the  unit  and  it  is  mandatory  that  soldiers  complete  the  questionnaire.  Data  collection 
usually  only  lasts  a  few  days,  since  the  commander  can  set  up  a  time  when  the  survey  is 
available.  Response  rates  are  not  known,  since  all  data  is  kept  at  the  unit  level. 
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One-Time  Attitude  Surveys 


Gallup  also  reviewed  three  “special”  surveys  conducted  by  ARI  researchers,  including  the 
Survey  of  Army  Families,  the  Officer  Evaluation  Report  Survey,  and  the  Survey  of  New  Recruit 
Values. 


The  Survey  of  Army  Families  gathers  information  on  the  attitudes  and  opinions  of  civilian 
spouses  of  soldiers  on  a  range  of  issues  such  as  family  support  services,  deployments  and  family 
separations,  housing,  and  other  issues.  It  has  been  conducted  three  times  so  far,  m  1987, 1991, 
and  1995,  and  may  be  conducted  again  next  year.  One  challenge  for  this  survey  is  that  it  is 
difficult  to  get  addresses  for  spouses.  The  next  round  will  test  using  the  Defense  Enrollment 
Eligibility  Registration  System  (DEERS)  to  get  spousal  addresses.  The  sample  size  in  1995  was 
20,000,  with  supplemental  oversamples  of  male  spouses  and  spouses  in  Europe,  pushing  the 
sample  size  up  to  33,000.  The  questionnaire  is  generally  20-24  pages  long  and  includes  items 
from  previous  iterations  of  the  survey  as  well  as  new  items,  which  are  pretested.  The  scannable 
survey  is  mailed  directly  to  home  addresses  when  they  are  available,  and  otherwise  are  mailed  to 
the  unit  address  of  the  soldier.  The  2000  survey  may  use  prenotification  (in  which  members  of 
the  sample  receive  an  advance  letter  describing  the  survey);  in  addition,  it  may  include  a 
reminder  notice  to  prod  sample  members  to  complete  the  survey.  The  response  rate  in  1995  was 
45%.  Data  are  weighted  by  rank  and  geographic  location  of  soldier. 


The  Survey  of  New  Recruit  Values  is  a  new  survey  that  is  establishing  a  database  of 
information  on  values  of  new  recruits  as  they  begin  their  Initial  Entry  Training.  Many  of  the 
items  were  borrowed  from  other  surveys  and  from  the  values  literature.  There  are  also  vignettes 
on  the  survey  to  try  to  gauge  how  new  recruits  would  respond  to  various  value-driven  situations. 
The  questionnaire  is  administered  in  a  group  setting  as  part  of  the  recruit’s  in-processing,  thus 
there  is  no  response  rate  issue  involved. 


The  Officer  Evaluation  Report  Survey  (OER)  assesses  officers’  attitudes  towards  specific 
procedures  being  considered  for  the  new  OER.  Most  of  the  questions  have  been  used  in  other 
surveys.  When  the  issues  have  not  changed,  the  survey  might  include  questions  from  as  many  as 
10  to  15  years  prior.  The  questionnaire  takes  about  15  to  20  minutes  to  fill  out,  and,  like  the 
SOC  questionnaire,  is  mailed  directly  to  the  officer’s  home  address  from  the  Officer  Master  File. 
The  data  collection  period  for  this  survey  is  relatively  short,  lasting  only  4-5  weeks,  but  can  go 
longer  depending  on  the  customer’s  needs.  The  response  rate  (50%)  was  lower  than  the  rate 
expected  (65%),  given  the  topic  and  the  audience.  Officers  may  have  believed  that  the  decisions 
were  already  made  and  their  input  would  be  ignored. 


Occupational  Analysis  Surveys 

The  purpose  of  Occupational  Analysis  research  is  to  provide  customers  rapid,  clear 
information  to  ensure  that  the  right  soldiers,  with  the  appropriate  skills,  are  available  to  meet 
Army- wide  requirements.  These  surveys  are  fully  automated.  Disks  are  mailed  either  in  bulk  or 
directly  to  the  individual,  and  the  disk  contains  the  electronic  questionnaire.  The  data  collection 
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period  is  lengthened  by  the  use  of  bulk  mail  to  distribute  the  disks.  There  are  tracking  numbers 
on  the  diskettes  so  that  follow-up  efforts  can  be  made  if  response  rates  are  low.  The 
questionnaires  take  about  20-30  minutes  to  complete.  The  items  about  the  tasks  that  make  up  a 
soldier’s  job,  and  the  same  scales  (which  assess  the  frequency  that  tasks  are  performed  in  doing  a 
given  job)  are  used  across  all  Military  Occupational  Specialties  (MOS).  If  the  MOS  includes 
more  than  1,000  people,  a  sample  is  drawn,  stratified  on  skill  level.  If  the  MOS  includes  fewer 
than  1,000,  all  of  them  are  included  in  the  sample.  Response  rates  are  generally  45-55%. 

Appendix  A  to  the  report  presents  more  detailed  information  about  each  of  these  ARI 
surveys. 
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BEST  practices  in  the  survey  industry 

Our  second  task  was  to  gather  similar  information  about  attitude  and  opinion  surveys  done 
by  private  and  government  organizations  outside  of  ARI.  In  consultation  with  ARI,  Gallup 
identified  11  surveys  and  obtained  information  about  their  practices  for  questionnaire  design  and 
construction  their  clearance  procedures,  as  well  as  their  sampling  methods,  data  collection, 
response  rates,  data  processing,  data  analysis,  and  reporting.  For  each  survey  we  investigated  we 
contacted  the  survey’s  research  director  or  project  director  and  administered  a  semi-structured 
interview.  Appendix  B  contains  more  detailed  information  about  each  survey.  The  key  findings 
of  Gallup’s  survey  of  the  field  mostly  involved  questionnaire  development,  response  rates  and 
coverage  (the  ability  to  obtain  opinions  from  all  members  of  the  target  population),  the  use  of 
computer  technology,  and  the  mode  of  survey  data  collection. 

The  surveys  that  were  selected  for  investigation  represent  a  variety  of  research  methods 
and  objectives.  Gallup  looked  at  the  General  Social  Survey  conducted  by  the  National  Opinion 
Research  Center  (NORC)  and  the  National  Election  Studies  conducted  at  the  University  of 
Michigan  as  “top  of  the  line”  attitude  studies.  These  studies  are  long-standing,  well-funded,  and 
highly  respected  within  the  survey  community.  Next,  as  a  comparison  to  the  occupational 
analysis  ARI  conducts,  Gallup  looked  at  occupational  analyses  conducted  by  PDRI.  Gallup  also 
investigated  the  research  practices  within  the  U.S.  Department  of  Defense.  We  collected 
information  on  the  Air  Force  Personnel  Survey  Program,  the  Navy-Wide  Personnel  Survey,  and 
the  Youth  Attitude  Tracking  Survey.  Two  national  polls  were  included,  the  Gallup  Poll  and 
Harris  Poll,  because  of  their  quick  turnaround  and  tight  timelines.  Finally,  two  organizational 
climate  studies  were  included,  the  Gallup  Workplace  Audit  and  the  Organizational  Climate  in 
Federal  Statistical  Agencies  conducted  at  the  University  of  Maryland.  These  studies  have 
purposes  similar  to  the  Army  Command  Climate  surveys. 


Questionnaire  Design 

If  a  questionnaire  fails  to  accurately  capture  what  it  is  intended  to  measure,  all  of  the 
efforts  made  in  sampling,  data  collection,  and  analysis  are  wasted.  As  a  result,  most  survey 
organizations  conduct  some  form  of  pretesting  as  part  of  the  questionnaire  development  process 
to  determine  the  effectiveness  of  a  particular  question  or  set  of  questions.  Gallup  found 
pretesting  to  be  a  common  practice  in  the  survey  organizations  we  examined,  except  where  quick 
turnaround  is  required.  Pretesting  in  these  organizations  was  done  using  focus  groups,  cognitive 
laboratories,  small  pilot  studies,  or  add-on  to  omnibus  studies. 

Another  step  that  organizations  have  taken  to  assess  the  validity  and  reliability  of  items  is 
to  compare  them  with  external  records  or  aggregate  benchmarks.  The  inclusion  of  questions  from 
previous  surveys  is  another  common  practice  in  the  survey  organizations  we  examined.  Using 
questions  from  other  surveys  allows  for  comparability  with  the  past  data  and  gives  the  researcher 
questions  that  have  been  “battle-tested”.  Several  of  the  surveys  used  standardized  scales  at  times 
to  provide  comparability  over  time  within  surveys;  the  use  of  scales  also  gives  respondents 
practice  in  answering  these  items,  which  reduces  burden.  Standardized  scales  also  lend 
themselves  to  combining  items  and  constructing  indices  when  conducting  analyses. 
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While  the  input  is  often  informal,  most  of  the  studies  Gallup  investigated  invite  customer 
input  at  the  questionnaire  design  stage.  Suggestion  boxes,  surveys,  and  formal  meetings  are  all 
methods  used  to  include  customers.  The  Youth  Attitude  Tracking  Survey,  which  meets  with 
members  of  each  branch  of  the  military  when  constructing  its  questionnaire,  is  a  good  example 
of  a  survey  that  involves  customers  in  the  design  of  the  research  instrument.  Customer  input  at 
the  design  stage  can  improve  the  content  and  utility  of  a  survey  and  help  the  researcher  set 
priorities  for  areas  of  investigation. 


Response  Rates  and  Coverage 

Omissions  due  to  nonresponse  and  coverage  problems  can  interfere  with  attempts  to  obtain 
representative  samples.  Every  sample  that  is  drawn  for  use  in  a  survey  must  come  from  a  list,  or 
sampling  frame,  of  some  kind.  Ideally,  these  lists  would  contain  the  names  of  every  person  that 
is  a  member  of  the  target  population  group.  However,  these  lists  often  omit  members  of  the 
population  or  contain  incorrect  information,  producing  coverage  errors. 

Our  investigation  of  the  field  found  such  frame  problems  to  be  very  common.  A  major 
reason  is  that,  in  both  organizations  and  households,  people  tend  to  be  mobile.  This  is  especially 
true  in  the  Department  of  Defense.  The  best  way  to  reduce  the  resulting  problems  in  locating 
members  of  the  sample  is  to  keep  up-to-date  records  of  the  location  of  personnel.  Within 
organizations,  payroll  records  are  a  good  source  of  up-to-date  information,  as  employees  are 
highly  motivated  to  keep  them  current. 

Coverage  can  also  be  related  to  the  choice  of  mode.  Telephone  surveys  fail  to  include 
persons  without  telephones  and  likewise  e-mail  surveys  omit  persons  who  do  not  have  access  to 
e-mail.  A  common  practice  in  the  design  of  organizational  surveys  is  to  conduct  analyses  to 
determine  the  level  of  coverage  a  particular  mode  will  provide.  Allowing  respondents  to 
participate  by  more  than  one  method  is  another  way  that  researchers  can  reduce  coverage  error. 
However,  if  multiple  modes  are  employed,  this  may  introduce  differences  due  to  the  mode  of 
interview',  these  mode  effects  may  reflect  differences  in  formatting,  communication  pathway 
(aural  vs.  visual),  and  perceived  confidentiality.  Establishing  linking  rules  that  systematically 
include  those  who  otherwise  would  not  be  captured  by  a  particular  mode  is  another  strategy  for 
coping  with  coverage  problems. 

The  omission  of  sample  members  through  nonresponse  is  another  common  concern. 
Researchers  seek  to  control  the  impact  of  nonresponse  on  the  accuracy  of  survey  estimates  by 
making  sure  that  response  rates  are  as  high  as  possible.  However,  response  rates  appear  to  be 
declining  throughout  the  survey  research  industry  (see,  for  example.  Brick  et  al.,  1997).  Survey 
research  organizations  may  seek  to  counteract  this  tendency  by  improving  their  efforts  at 
persuasion  (for  example,  by  sending  prenotification  letters  and  offering  incentives  for 
participation)  and  by  making  more  contact  attempts.  Follow-up  efforts,  such  as  reminder  post¬ 
cards,  second  mailings  of  questionnaires,  and  increased  call  attempts  in  interviewer-administered 
modes  have  been  widely  adopted  to  combat  nonresponse.  In  organizational  settings,  distribution 
of  the  questionnaire  by  an  immediate  supervisor  often  yields  high  response  rates,  but  may  affect 
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how  people  answer  the  questions,  especially  any  sensitive  items.  Lengthening  the  field  period  to 
give  people  more  opportunity  to  respond  is  another  tactic  that  can  improve  response  rates. 


Mode  of  Data  Collection 


The  choice  of  mode  of  data  collection  can  affect  costs,  coverage  and  response  rates, 
timeliness,  and  the  answers  respondents  give.  Personal  interviews  tend  to  produce  the  highest 
response  rates  (as  in  the  General  Social  Survey  and  the  National  Election  Studies).  However,  the 
cost  of  face-to-face  interviewing  can  be  prohibitive.  Mail  and  telephone  surveys  tend  to  yield 
lower  response  rates  than  those  obtained  in  personal  interviews,  but  are  often  used  because  they 
are  much  more  affordable.  Some  of  the  modes  recently  adopted  by  survey  organizations  include 
touch-tone  data  entry  and  voice  recognition  which  allow  a  person  to  respond  by  phone  without  a 
live  interviewer,  and  e-mail,  Web,  and  disk-by-mail  surveys,  which  allow  respondents  to  answer 
at  their  computer.  These  modes  are  mostly  used  in  organizational  settings  where  computers  are 
prevalent.  The  response  rates  tend  to  vary  with  the  effort  by  organizations  to  get  people  to 
respond.  Mixed  mode  and  multiple  mode  surveys  are  common  when  interviewmg  employee 
populations,  but  systematic  differences  by  mode  in  the  answers  are  sometimes  found. 

Personal  interviews  and  mail  surveys  tend  to  require  longer  field  periods  than  telephone  or 
e-mail  surveys.  Generally,  it  takes  longer  to  contact  the  sample  member  m  person  or  by  mail 
than  by  telephone  or  e-mail.  Telephone  interviewing  also  offers  the  advantage  of  central 
supervision  of  interviewers  as  compared  to  the  supervision  available  in  a  face-to-face  survey. 
Telephone  interviewers  can  be  randomly  monitored  without  their  knowledge  whereas  fie  d 
interviewers  are  aware  when  their  supervisor  is  present  and  may  modify  their  behavior.  When 
sensitive  data  are  to  be  collected,  self-administered  modes  are  usually  preferred  to  modes  in 
which  interviewers  administer  the  questions  since  self-administration  has  been  shown  to  increase 
respondent  candor. 


Computer-assisted  modes  of  data  collection  offer  some  advantages  over  paper-and-pencil 
methods  and  have  been  adopted  for  most  of  the  non-ARI  surveys  we  investigated.  In 
administering  surveys,  computer  assistance  ensures  that  complex  skip  patterns  are  followed 
exactly  and  that  question  options  are  rotated  appropriately;  in  addition,  it  allows  the  length  of 
time  spent  answering  each  item  to  be  measured.  After  the  data  have  been  collected, 
computerization  reduces  the  time  and  expense  of  coding  the  data  and  processing  them  to  compile 
a  usable  data  set.  Even  when  mail  questionnaires  were  used  in  the  surveys  we  exammed,  they 
tended  to  use  scannable  questionnaires  to  speed  data  entry  and  processing. 


Summary 

Appendix  B  presents  additional  details  on  the  1 1  non-ARI  surveys  we  reviewed.  Our 
survey  of  the  field  found  that  pretesting  the  questions  is  a  common  practice  when  designing 
surveys.  Customer  input  is  also  commonly  invited  by  organizations  as  they  design 
questionnaires.  ARI  obtains  response  rates  that  are  comparable  to  other  organizations  that  use 
similar  modes  of  data  collection,  but  ARI  may  be  lagging  somewhat  in  the  adoption  of  computer- 
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assisted  data  collection  technologies.  Although  mail  and  telephone  surveys  predominated  in  our 
sample,  e-mail,  Web,  and  automated  telephone  surveys  are  becoming  more  common,  especially 
in  organizational  settings,  where  multiple  modes  are  often  used  to  collect  data. 
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CUSTOMER  SATISFACTION 


Gallup’s  third  task  was  to  assess  customer  satisfaction  with  ARI’s  programs  and  services. 
This  effort  was  the  first  time  ARI  had  systematically  interviewed  its  customer  base  to  determine 
the  extent  that  current  programs  were  meeting  their  needs.  Gallup  compiled  both  qualitative  and 
quantitative  data  from  a  list  of  customers  created  for  this  project. 


ART’s  Customers 

A  current  list  of  ARI  customers  did  not  exist  at  the  time  of  the  study.  Gallup  interviewed 
key  ARI  survey  staff  to  compile  a  list  of  customers.  In  addition,  Gallup  staff  contacted  each 
customer  individually  to  verify  their  relation  to  ARI  and  to  ask  them  to  offer  names  of  other  ARI 
customers  The  final  list  totaled  39  customers  from  a  wide  range  of  command,  personnel,  and 
education  departments  within  the  United  States  Army.  To  protect  the  confidentiality  of  the 
respondents,  Gallup  cannot  list  the  names  of  those  who  took  part  in  the  study.  However,  a  partial 
list  of  their  agencies  is  provided  below: 

•  Army  Community  and  Family  Support  Center 

•  Army  Safety  Center 

•  Commanders  of  company-sized  units 

•  Office  of  the  Deputy  Chief  of  Staff  for  Personnel  (DCSPER) 

•  Secretary  of  the  Army 

•  US  Army  Training  and  Doctrine  Command  (TRADOC) 

•  United  States  Military  Academy 


Eliciting  Feedback 

We  used  the  customer  list  to  recruit  individuals  to  participate  in  the  qualitative  and 
quantitative  research  efforts.  We  organized  a  focus  group  of  five  customers  held  at  an 
independent  research  facility  in  Alexandria,  Virginia.  The  goal  of  the  focus  group  was  to 
identify  the  issues  and  concerns  voiced  by  customers  as  they  related  their  experiences  with  ARI. 
Questions  posed  to  the  participants  were  intentionally  open-ended,  to  allow  customers  to  talk 
freely  about  their  working  relationship  with  ARI  programs  and  staff.  The  discussion  was  audio- 
taped  to  assist  the  writing  of  the  report  summarizing  the  discussion. 

Following  the  qualitative  session,  Gallup  collected  survey  data  by  telephone  with  31  of  the 
remaining  34  ARI  customers.  The  interviewers  who  called  ARI’s  clients  were  Gallup’s 
executive  interviewers,  a  special  group  of  professionals  who  are  particularly  experienced  in 
surveying  top-level  managers  in  the  public  and  private  sector. 

The  instrument  was  based  upon  Gallup’s  own  standard  survey  for  assessing  the  satisfaction 
of  its  research  customers.  Gallup  researchers  get  quarterly  ratings  on  how  satisfied  their 
customers  are  with  their  work  and  the  company’s  work  as  a  whole.  The  questionnaire  focuses  on 
both  general  satisfaction  (e.g.,  likelihood  of  recommending  the  product  or  service  to  another 
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potential  customer,  overall  value)  as  well  as  specific  factors  leading  to  satisfaction  (e.g. 
responsiveness  to  customer  requests,  ability  to  uncover  insights). 


The  standard  Gallup  Researcher’s  Survey  was  modified  to  cover  specific  ARI  programs 
and  services,  as  well  to  take  advantage  of  knowledge  gained  from  the  focus  group.  For  example, 
an  issue  that  was  raised  in  the  focus  group  was  a  question  of  how  customers  rated  different 
components  of  ARI  compared  to  other  organizations.  As  a  result,  particular  attention  was  paid  to 
the  questions  on  ARI’s  general  reputation  and  compared  to  other  research  organizations  with 
which  they  may  have  worked. 


Focus  Group  Results 

Three  out  of  five  of  the  respondents  were  almost  exclusively  current  ARI  customers  of  the 
SSMP.  One  respondent  reported  use  of  a  Command  Climate  survey.  Other  interactions  with 
ARI  included  reviews  of  questions  for  unit  command  surveys,  need  assessments,  and  surveys  o 
customer  satisfaction  with  Post  Exchange  facilities.  In  addition  to  ARI  reports,  all  respondents 
but  one  used  data  from  other  sources,  such  as  Defense  Manpower  Data  Center  (DMDC) 
population  statistics  and  the  Officer  Master  File.  Cost  and  expertise  were  cited  as  the  two  mam 
factors  motivating  the  decision  to  use  ARI  or  a  private  contractor.  More  often  than  not,  ARI  was 
seen  as  more  cost-effective  than  an  outside  contractor  but,  in  some  cases,  was  not  seen  as  a  goo 
fit  for  a  particular  study.  Size  of  the  survey  was  also  seen  as  a  factor,  with  customers  preferring 
ARI  to  handle  large-scale  projects. 


The  focus  group  respondents  said  they  want  ARI  to  be  responsive  to  their  current  issues 
and  needs.  When  customers  want  to  have  specific  topics  covered,  they  must  submit  a 
justification  form  to  ARI.  Clients  expect  the  analyses  to  be  accurate  and  communicated  in  a  way 
that  can  be  easily  understood.  In  addition,  they  would  like  to  see  the  analyses  compared  to  other 
sources  over  time,  placing  the  results  in  the  larger  context.  ARI  should  be  proactive  in  handling 
problems  with  its  clients,  such  as  asking  them  if  they  can  afford  to  deal  with  negative  responses 
to  proposed  questions.  Electronic  report  distribution  was  seen  as  very  desirable,  and  ARI  was 
encouraged  to  widely  publicize  the  location  of  the  files.  In  general,  ARI’s  analyses  were  ^ 
comprehensive  and  clear;  its  reports  were  not  cluttered  with  excessive  technical  detail.  ARI  s 
presentations  clarified  the  material  in  the  reports  very  well. 

Focusing  on  customer  satisfaction,  we  asked  the  participants  to  identify  the  most  important 
components  of  the  service  they  receive  from  ARI.  The  focus  group  respondents  mentioned 
many  of  the  same  components  assessed  in  the  standard  Gallup  survey  of  its  customers  described 
earlier.  The  following  list  of  issues  related  to  customer  satisfaction  emerged  from  the  focus 
group: 

•  Questionnaire  design; 

•  Responsiveness; 

•  Availability  of  ARI  staff; 

•  Attitude  of  the  staff; 

•  Overall  quality  of  staff ; 
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•  Quality  of  the  analysis; 

•  Communicating  analysis  effectively; 

•  Meeting  their  needs; 

•  Timeliness. 

Respondents  said  there  was  not  a  standard  mechanism  in  place  for  them  to  provide  ARI 
feedback.  However,  feedback  was  given  throughout  the  planning  and  implementation  of  the 
study.  Overall  feedback  was  usually  given  at  the  final  briefing  face-to-face. 


Customer  Survey  Results 

In  examining  the  survey  results,  Gallup  first  computed  overall  mean  scores  for  each 
question.  ARI  scored  very  well  on  both  overall  and  specific  measures  of  customer  satisfaction; 
the  mean  scores  did  not  vary  markedly  across  the  questions.  To  evaluate  specific  areas  of 
strength,  we  analyzed  “top  box”  scores— the  proportion  of  customers  giving  ARI  the  best  rating 
available  on  a  particular  item.  Most  of  the  questions  employed  a  five-point  scale,  with  “very 
satisfied”  as  the  highest  response  category. 

To  assist  in  understanding  the  results  from  the  20  satisfaction  items  in  the  questionnaire,  it 
is  useful  to  talk  about  general  themes  in  the  data.  We  conducted  a  factor  analysis  to  help 
organize  and  simplify  the  analysis.  Factor  analyses  are  often  useful  in  finding  common  themes 
in  data,  providing  a  framework  for  action  as  well  as  interpretation.  Because  of  the  small  sample 
size  in  the  present  case,  the  factors  that  emerged  should  be  seen  as  tentative  rather  than 
definitive.  Nonetheless,  the  results  seemed  quite  interpretable,  indicating  that  the  satisfaction 
items  fell  into  three  mam  groups; 

Front-end  items— items  on  the  services  ARI  provides  to  its  clients  prior  to 
fielding  the  survey; 

Back-end  items — questions  on  services  ARI  provides  after  the  data  are  collected; 
Relationship  items — questions  on  how  ARI  interacts  with  its  clients. 

Because  ARI’s  scores  were  relatively  high  on  all  20  items,  we  made  an  arbitrary  distinction 
between  top  box  scores  that  were  equal  to  or  above  60%  and  those  scores  that  were  below  60%. 
The  former  are  described  as  ARI’s  “best”  strengths,  while  those  below  the  line  are  pictured  as 
“good”  scores. 

ARI’s  highest  scores  came  in  the  Relationship  category.  Seventy-seven  percent  of 
customers  interviewed  said  they  were  very  satisfied  with  ARI’s  ability  to  work  on  a  friendly  and 
cooperative  basis,  while  65%  were  also  very  satisfied  with  ARI’s  overall  responsiveness  to  client 
needs.  Contributing  to  these  overall  ratings  are  the  outstanding  scores  that  individual  researchers 
at  ARI  receive  from  their  clients.  Again,  77%  of  customers  said  they  were  very  satisfied  with 
the  quality  of  service  they  received  from  their  ARI  researcher.  These  results  should  be 
underscored.  Service  organizations  do  not  necessarily  produce  tangible  products  for  their  clients 
very  often.  Therefore  it  is  crucial  for  research  organizations,  like  ARI,  to  maintain  strong  client 
relations.  It  appears  that  ARI  has  excelled  in  this  area. 
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ARI’s  top  box  scores  were  also  very  good  on  front-end  issues,  such  as  sampling  procedures 
and  survey  instrument  development  (62%).  On  the  back-end,  ARI  scored  particularly  well  on 
the  accuracy  of  cross-tabulations  (67%),  digging  through  the  data  to  uncover  insights  (60%),  and 
the  in-person  presentation  of  the  data  by  staff  (60%).  One  area  where  ARI  could  improve  is  the 
overall  appearance  of  report  documents,  which  garnered  a  relatively  low  top  box  score  of  50%. 

On  the  overall  satisfaction  measures,  ARI  again  gets  high  ratings  from  its  clients.  Sixty- 
eight  percent  of  ARI’s  customers  gave  a  Top  Box  score  for  the  value  of  the  products  and  services 
they  received.  Almost  two-thirds  (65%)  of  ARI’s  customers  said  they  were  very  satisfied 
compared  to  39%  for  the  same  question  when  asked  of  organizations  other  than  ARI  that  do 
similar  work.  Moreover,  seventy-one  percent  said  they  would  recommend  ARI  to  a  colleague. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


Overall  Performance 

Our  review  indicates  that  ARI  is  doing  a  good  job.  It  carries  out  surveys  at  a  relatively  low 
cost,  uses  methodologically  sound  procedures,  and  achieves  reasonably  high  response  rates.  It 
has  moved  with  the  times  and  is  increasingly  relying  on  computer-assisted  data  collection. 

ARI’s  customers  at  a  wide  range  of  agencies  find  the  data  useful  and  are  highly  satisfied  with 
ARTs  work.  For  example,  nearly  two-thirds  (65%)  of  the  respondents  in  the  customer 
satisfaction  survey  we  conducted  said  they  were  very  satisfied  with  ARI’s  overall  performance. 
ARI’s  surveys  exhibit  the  same  sorts  of  problems  (such  as  response  rate  and  coverage  problems) 

as  the  benchmark  surveys  we  examined. 

Based  on  our  review  of  ARI’s  current  practices,  our  customer  satisfaction  survey,  and  our 
benchmarking  study  of  comparable  surveys  done  by  other  organizations,  we  present 
recommendations  in  six  areas — response  rates,  pretesting,  new  survey  technologies,  timeliness, 
deliverables,  and  customer  feedback. 


Recommendation  1:  Response  Rates 

It  is  not  entirely  clear  what  the  appropriate  benchmark  is  for  evaluating  the  response  rates 
obtained  in  ARI’s  surveys.  On  the  one  hand,  ARI’s  response  rates  are  quite  high  given  the  basic 
method  of  contact  (mail)  and  the  minimal  follow-up  effort  in  several  of  the  surveys.  The 
literature  on  mail  surveys  is  very  clear  about  the  importance  of  follow-up  efforts  such  as 
reminder  postcards  and  replacement  questionnaires — in  achieving  high  response  rates  (e.g., 
Heberlein  &  Baumgartner,  1978).  In  addition,  the  response  rates  at  ARI  are  generally  in  line 
with  those  achieved  by  the  benchmark  surveys  we  examined.  On  the  other  hand,  the  response 
rates  are  somewhat  lower  than  might  be  expected  given  the  “captive”  population  that  ARI 
generally  surveys.  Further,  one  unexpected  finding  from  our  review  of  ARI’s  surveys  is  that  a 
substantial  component  of  the  nonresponse  in  these  surveys  results  from  the  failure  to  contact  the 
sample  members  rather  than  their  unwillingness  to  take  part  once  ARI  has  reached  them.  This 
reflects  the  reliance  on  questionnaires  mailed  to  the  sample  member’s  unit  address;  unit  address 
information  is,  apparently,  sometimes  inaccurate. 

We  recommend  several  steps  to  address  these  concerns  about  response  rates.  First,  we 
recommend  that  ARI  undertake  studies  to  identify  the  correlates  of  nonresponse  and  to  examine 
in  more  detail  the  impact  of  incorrect  locating  information  on  the  Army  Enlisted  Master  Files. 
(This  is  the  frame  used  for  several  ARI  studies.)  The  study  on  nonresponse  might  involve 
comparing  response  rates  for  officers  vs.  enlisted  personnel,  for  soldiers  stationed  on  the 
continental  U.S.  vs.  those  stationed  elsewhere,  as  well  as  examining  differences  in  response  rates 
by  rank,  education,  type  of  command,  and  length  of  service.  The  purpose  of  identifying  these 
correlates  of  nonresponse  is  to  develop  hypotheses  about  steps  to  improve  the  response  rates. 

For  example,  the  results  of  an  analysis  of  nonresponse  might  suggest  tailoring  a  cover  letter  to 
persuade  specific  subgroups  of  the  importance  of  their  participation. 
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The  study  of  the  bias  from  out-of-date  locating  information  on  the  Army  Enlisted  Master 
Files  would  help  determine  whether  it  is  worth  the  cost  to  try  to  improve  the  frame  or  to  move  to 
a  different  method  of  data  collection.  These  costs  are  likely  to  be  high  and  would  not  be  justified 
unless  the  bias  resulting  from  the  omission  of  those  with  poor  address  information  is  substantial. 
It  is,  therefore,  useful  to  estimate  the  magnitude  of  the  bias  by  comparing  the  characteristics  of  a 
small  sample  of  these  nonrespondents  with  a  sample  of  soldiers  whose  unit  address  information 
is  accurate. 

The  following  additional  recommendations  are  intended  to  increase  response  rates.  We 
believe  that  ARI  is  likely  to  continue  to  use  mail  questionnaires  as  its  main  method  of  data 
collection  for  the  foreseeable  future.  This  methodology  has  proved  to  be  a  very  cost-effective 
means  for  obtaining  information.  Several  steps  may  help  improve  response  rates.  We 
encourage  ARI  to  experiment  to  fmd  the  best  combination  of  changes  to  improve  response  rates. 
These  changes  might  include: 

•  Sending  a  prenotification  letter  to  sample  members; 

•  Developing  more  “respondent-friendly”  questionnaires  (that  is,  questionnaires 
that  are  easier  to  understand  and  complete); 

•  Improving  the  cover  letter  included  with  the  questionnaire; 

•  Providing  some  sort  of  incentive  with  the  questionnaire  (although  a  monetary 
incentive  may  be  inappropriate  in  this  context,  a  summary  of  highlights  of  prior 
findings  might  boost  response  rates); 

•  Adding  such  forms  of  follow-up  as  replacement  questionnaires  or  repeated 
reminder  postcards. 

In  the  past,  ARI  has  mailed  questionnaires  with  a  preassigned  id  number,  but  the  questionnaire 
itself  has  been  anonymous.  The  anonymity  of  the  questionnaires  has  prevented  any  follow-up 
efforts  directed  solely  at  nonrespondents  (since  ARI  cannot  track  who  has  responded).  It  seems 
likely,  however,  that  the  preassignment  of  an  id  number  to  the  questionnaire  has  compromised 
the  perception  of  anonymity.  We  think  it  is  important  that  most  ARI  surveys  remain  anonymous 
(to  encourage  honest  answers),  so  that  we  recommend  eliminating  the  preassigned  id  numbers. 
We  also  believe  that  some  minimal  follow-up  efforts  are  likely  to  yield  increased  response  rates. 
Respondents  could  be  asked  to  return  a  card  separate  from  the  questionnaire  itself  in  which  they 
identify  themselves  and  indicate  that  they  have  responded.  This  would  allow  reminder 
postcards  and  replacement  questionnaires  to  be  sent  only  to  the  initial  nonrespondents,  without 
compromising  anonymity.  Or  appropriately  written  reminder  postcards  could  be  sent  to  all 
sample  members.  Whatever  form  it  takes,  some  form  of  follow-up  is  likely  to  be  needed  to 
improve  response  rates. 


Recommendation  2:  Pretesting 

We  have  already  mentioned  the  importance  of  questionnaire  design  in  our  previous 
recommendation.  Currently,  there  does  not  seem  to  be  a  general  policy  toward  pretesting  at  ARI 
or  a  standardized  procedure  for  testing  and  evaluating  new  survey  items.  As  a  result,  items  that 
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have  not  been  pretested  are  often  included  in  ARI  surveys.  In  addition,  many  items  that  are 
routinely  included  in  ARI’s  recurrent  surveys  are  probably  due  for  reevaluation.  For  example, 
many  of  the  questionnaires  we  reviewed  include  items  that  use  a  check-all-that-apply  format, 
such  items  have  characteristic  problems,  often  leading  to  underreporting  (Rasmski,  Mingay,  & 
Bradburn,  1994). 

Based  on  our  assessment  of  ARI  questionnaires,  we  recommend  that  ARI  improve  and 
standardize  its  pretesting  procedures.  More  specifically,  we  recommend  that  new  items  undergo 
cognitive  testing  with  nine  or  ten  respondents.  Similarly,  we  recommend  that  small-scale 
usability  testing  be  done  on  automated  questionnaires.  Finally,  items  that  have  been  used  m  the 
past  should  be  subjected  to  periodic  reevaluation  via  cognitive  interviewing  or  reviews  by  expert 
panels  to  ensure  they  are  still  appropriately  measuring  the  variable  of  interest.  When 
questionnaires  are  being  developed  on  new  topics,  these  pretesting  procedures  should  be 
supplemented  with  focus  groups  or  ethnographic  interviews  to  develop  an  understanding  of  how 
members  of  the  target  population  think  and  talk  about  the  subject  under  investigation.  Expert 
review  panels  are  another  low-cost  method  for  improving  questionnaires  and  might  be  adopted 
as  part  of  the  standard  procedure  for  developing  new  items  or  new  questionnaires. 


Recommendation  3:  New  Survey  Technologies 

ARI  has  already  begun  to  incorporate  new  data  collection  technologies — such  as  disk-by¬ 
mail  (used  in  the  occupational  analysis  surveys)— in  its  data  collection  efforts.  We  offer  several 
recommendations  for  accelerating  the  adoption  of  these  new  tools. 

First,  we  recommend  ARI  adopt  some  of  the  newest  data  collection  tools,  such  a„  ” 
surveys,  e-mail,  and  interactive  voice  recognition  (in  which  digitized  recordings  of  the  questions 
are  administered  by  telephone  and  the  respondent  answers  questions  by  pressing  keys  on  the 
keypad  of  a  touchtone  phone).  Several  of  these  data  collection  methods  may  be  particularly 
well-suited  for  collecting  data  from  subpopulations  (such  as  officers)  with  ready  access  to 
computers  or  telephones.  We  note,  however,  that  the  transition  to  these  new  modes  of  data 
collection  may  require  a  substantial  testing  effort  to  determine  whether  the  mode  itself  alters  the 
answers  to  the  questions.  Second,  we  recommend  that  ARI  investigate  the  use  of  more 
sophisticated  scanning  equipment.  Currently,  ARI’s  mail  surveys  require  respondents  to  indicate 
their  answers  by  filling  in  the  appropriate  bubbles.  More  sophisticated  scanning  equipment 
would  permit  greater  flexibility  in  the  method  of  responding;  for  example,  some  scanners  can 
recognize  characters.  Although  the  new  scanning  technologies  are  expensive,  prices  are  likely  to 
fall.  Finally,  we  recommend  that  ARI  continue  to  adopt  technologies  that  permit  the  automation 
of  other  survey  operations,  such  as  the  generation  of  codebooks  or  standardized  reports. 


Recommendation  4:  Timeliness 

One  of  the  major  challenges  that  ARI  faces  is  to  provide  timely  data  to  help  settle  emerging 
policy  issues.  Unfortunately,  several  obstacles  may  prevent  ARI  from  getting  results  quickly 
enough  to  influence  the  decisions  that  need  to  be  made.  For  example,  the  mail  methods  ARI 
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typically  employs  do  not  always  permit  fast  turnaround.  (In  addition,  when  clearance  from  the 
Office  of  Management  and  Budget  or  DOD  is  needed,  data  collection  may  be  delayed  for  as  long 
as  a  year.)  These  delays  may  not  affect  the  usefulness  of  the  results  from  ARI’s  continuing 
studies,  but  they  can  reduce  the  impact  of  the  findings  on  new  or  pressing  issues. 

To  address  this  concern,  we  recommend  that  ARI  explore  methods  to  create  a  fast 
turnaround  capability.  One  method  might  be  to  recruit  a  master  sample  including  5,000  to 
10,000  cases  at  the  beginning  of  each  year.  A  few  basic  pieces  of  information  would  be 
collected  from  the  members  of  this  sample,  most  crucially,  up-to-date  telephone  numbers  and 
mailing  and  e-mail  addresses.  Then,  as  issues  arose  that  demanded  fast  returns,  subsamples  of 
the  master  sample  could  be  selected  and  interviewed  by  telephone  (or  in  some  cases  via  e-mail  or 
mail). 

Of  all  our  recommendations,  this  one  is  likely  to  involve  the  most  cost.  The  largest  costs 
would  be  in  developing  and  maintaining  an  up-to-date  list.  Thus,  the  need  for  timely  survey 
information  must  be  weighed  against  the  likely  cost  of  creating  a  fast  turnaround  capability. 


Recommendation  5:  Deliverables 


Like  many  survey  organizations,  ARI  tailors  its  products  to  the  customers.  One  outcome  of 
this  practice,  however,  is  that  there  is  no  standard  documentation  available  for  each  ARI  survey, 
covering  the  basic  methodology  it  employed,  the  response  rates  it  achieved,  and  other  standard 
topics.  We  recommend  that  a  minimal  methodology  report  be  routinely  produced  for  each  ARI 
study.  This  standard  report  should  cover  such  basic  issues  as  the  sampling  frame  for  the  study, 
the  sample  design,  the  questionnaire  used,  the  method  of  data  collection  and  field  period,  the 
response  rate,  and  any  special  procedures  used  in  developing  survey  estimates  (such  as  weights 
or  imputation  of  missing  values). 


ARI  is  in  the  process  of  archiving  many  of  its  existing  data  sets.  Our  recommendation  for 
producing  standard  documentation  complements  this  ongoing  effort. 


Recommendation  6:  Customer  Feedback 

Although  ARI’s  customers  are  currently  very  satisfied  with  ARI’s  surveys,  there  are  no 
formal  procedures  in  place  for  gathering  customer  feedback  on  a  systematic  basis.  Our  final 
recommendation  is  that  ARI  establish  some  mechanism  for  obtaining  regular  feedback  from  its 
customers.  This  might  take  any  of  several  forms.  For  example,  some  organizations  have  set  up 
advisory  panels  that  include  customers.  Others  conduct  formal  customer  satisfaction  surveys, 
like  the  one  we  used  in  this  project,  on  a  routine  basis.  Still  others  conduct  periodic  focus  groups 
with  customers.  Given  the  importance  of  customer  input  during  both  the  design  and  analysis  of 
surveys,  we  think  it  is  particularly  important  for  ARI’s  survey  units  to  understand  how  then- 
customers  view  them. 
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REVISED  REGULATION 

Our  final  task  was  to  review  the  existing  Array  policies  and  regulations  pertaining  to  the 
attitude  and  opinion  survey  programs,  and  to  prepare  drafts  of  updates  to  those  policies  and 
regulations.  The  attitude  and  opinion  survey  programs  is  described  in  Army  Regulation  600-46 
This  regulation  was  originally  published  in  November  of  1978,  with  an  interim  change 
implemented  in  April  of  1983.  Gallup  reviewed  AR  600-46  and  additional  supporting 
regulations,  policies  and  documents.  Recommendations  for  changes  to  AR  600-46  and  other 
documents  are  contained  in  this  section  and  Appendix  C  to  this  report.  These  final  draft 
regulations  and  policies  reflect  ARI’s  comments  on  the  draft  recommendations  submitted  by 

Gallup. 


Gallup  staff  interviewed  the  Chief  Army  Personnel  Survey  Officer  to  compile  a  list  of 
applicable  regulations,  policies  and  documents.  Additional  documents  were  identified  as  the 
review  process  continued.  The  list  of  documents  reviewed  is  provided  below: 

•  DOD  1100.13  Surveys  of  DOD  Personnel 

•  ARI  1-2  Management  Control 

•  AR  25-55  The  Department  of  the  Army  Freedom  of  Information  Act  Program 

•  AR  25-400-2  The  Modern  Army  Record  Keeping  System  (MARKS) 

•  AR  70-25  Use  of  Volunteers  as  Subjects  of  Research 

•  AR  340-21  Army  Privacy  Program  (superseded  by  AR  25-55) 

•  AR  570-4  Manpower  Management 

•  AR  6-8-104  Military  Personnel  Information  Management/Records 

•  AR  600-46  Attitude  and  Opinion  Survey  Program 

•  AR  611-3  Army  occupational  Survey  Program 

•  DA  Pam  600-8  Military  Personnel  Management  and  Administrative  Procedures 

•  ARI 70-26  Data  Collection  Instrument  Review  Policy 

•  ARI  70-30  Data  Sharing 

•  ARI  70-31  DRAFT  -  Data  Documentation  and  Archival  Policy 

•  5  CFR  1320  Controlling  Paperwork  Burdens  on  the  Public 

•  AH  36-2601  Air  Force  Personnel  Survey  Program 

•  OPNAV  5300.8B  Coordination  and  Control  of  Personnel  Surveys 


Document  Review  Results 

The  documents  were  cross-referenced  for  applicability  and  to  insure  compliance  with 
current  Army  policy.  AR  600-46  was  reviewed  to  determine  the  regulation’s  relevancy  and  its 
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appropriateness  to  current  ARI  requirements  and  practices.  Army  regulations  were  also 
compared  with  the  corresponding  Air  Force  regulations  and  Navy  instructions. 

The  published  version  of  AR  600-46,  with  its  interim  changes,  was  found  to  be  outdated 
and  inaccurate.  The  format  was  inconsistent  with  more  recently  published  regulations.  The 
regulation  placed  authorities  and  responsibilities  on  organizations  that  were  eliminated  or 
renamed  as  a  result  of  the  organizational  realignment  of  the  Army.  In  addition,  the  regulation 
was  found  to  not  adequately  reflect  ARI’s  current  requirements  and  practices. 

We  also  compared  Army,  Air  Force  and  Navy  regulations  for  the  control  and  management 
of  attitude  and  opinion  surveys.  Highlights  of  that  comparison  are  outlined  in  Table  1. 

TABLE  1.  MILITARY  SERVICES  COMPARISONS 


The  results  of  Gallup’s  qualitative  and  quantitative  data  analysis  and  the  resulting  findings 
and  recommendations  were  also  considered  during  the  regulation  review.  The  following  items 
are  not  addressed  in  the  published  regulation: 

•  Standardized  pretesting  of  survey  questions 

•  Guidelines  for  providing  deliverables  to  survey  sponsors 

•  Requirements  for  the  archiving  of  survey  results 

•  Guidelines  or  instructions  for  the  release  of  survey  findings 
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Recommendations 


Appendix  C  provides  a  draft  revision  of  AR  600-46.  Because  of  major  changes  in  the 
language  used,  and  the  content  of  this  revision,  no  attempt  has  been  made  to  reference  or 
highlight  changes  from  the  previous  regulation.  Input  was  solicited  from  ARI  staff  during  the 
revision  of  the  regulation.  The  draft  revision  incorporates  ARI’s  current  practices,  appropriate 
references  to  Army  agencies  and  regulations,  and  the  initial  Gallup  recommendations. 

Currently  the  only  guidelines  on  the  release  of  survey  data  are  those  identified  below.  The 
existing  law  generally  requires  that  federal  agencies  release  any  information  that  is  specifically 
identified  and  requested  under  terms  of  the  Freedom  of  Information  Act.  In  accordance  with  the 
published  regulation,  ARI  has  honored  the  stated  requests  of  survey  sponsors  (including  the 
sponsors  of  survey  items  included  in  the  Sample  Survey  of  Military  Personnel)  concemmg  the 
release  of  data.  The  options  for  the  survey  sponsors  are: 

•  To  release  data  to  any  requesters  or  media; 

•  To  release  data  to  requesters/media  other  than  those  specified  for  exclusion  (with 
the  reasons  for  exclusion  to  be  specified); 

•  To  not  release  data  to  anyone. 

Army  commands  or  agencies  and  DoD  agencies  are  the  most  frequent  requesters  of  data. 

In  addition,  the  news  media  (such  as  the  New  York  Times,  U.S.  News  and  World  Report,  and  the 
Knight-Ridder  news  service)  request  and  receive  data  summaries  and  data  files.  Survey  findings 
are  valuable,  and  to  increase  their  value  the  information  should  be  shared  with  Army 
Commanders;  still,  in  some  cases,  it  may  be  desirable  to  exempt  some  information  from  release 

to  the  media  or  even  other  military  organizations. 

In  addition,  Gallup  requested  specific  guidance  from  the  Senior  Army  Staff  on  release  of 
survey  data,  which  was  incorporated  into  the  draft  Army  and  ARI  regulations. 
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Task  1:  Review  of  ARI  Surve 
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Current  Methods  Used: 
Special  Surveys 
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Current  Methods  Used: 
Special  Surveys 
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Current  Methods  Used 
Occupational  Analyses 
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Task  2:  Survey  of  the  Field 


Current  Methods  Used 
Military  surveys 
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6.  Management  report,  trend  analysis,  and 
cross-tab  tables. 


Current  Methods  Used 
Military  surveys 
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5.  SPSS  for  the  Army  and  Navy  and  SAS  for 

Rand  and  Air  Force. 

6.  Branches  conduct  their  own  analyses. 


Current  Methods  Used 
Polling  organizations 
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trending  and  analytical  reporting  often 
done  as  well. 


I  Current  Methods  Used 
Organizational  Climate  Surveys 
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6.  Usually  descriptive.  May  do  driver  5.  SAS  wd  SPSS  versions  made  available. 

analyses  w/  linking  to  business  outcomes. 

6.  Agencies  conducted  their  own. 
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Computerized  methods  are  widely  adopted. 
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Chapter  1 
Introduction 


DRAFT 
AR  600-46 


1-1.  Purpose 

This  regulation — 

a.  Establishes  policy,  guidelines,  procedures,  definitions,  and  assigns  responsibilities  for 
authorization  and  coordination  of  personnel  surveys  conducted  of  Army  personnel. 

b.  Establishes  administrative  procedures  and  assigns  responsibilities  in  support  of  the 
following  Army  personnel  survey  programs  conducted  by  the  Army  Research  Institute 
(ARI) 

(1)  Sample  Survey  Program:  RCS:  MILPC-3. 

(2)  Special  Army  Personnel  Surveys:  RCS:  MILPC- 100. 

1-2.  References 

Required  and  related  publications  are  listed  in  appendix  A. 

1-3.  Explanation  of  abbreviations  and  terms 

Abbreviations  and  special  terms  used  in  this  regulation  are  explained  in  the  glossary. 
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Chapter  2 

Survey  Authorization  and  Coordination 
2-1.  Purpose 

Authorization  and  coordination  of  Army  personnel  surveys  will  facilitate- 

a.  Linkages  among  Army  personnel  proponents,  including  information  on  surveys  being 
planned  and  conducted. 

b.  Use  of  high  quality  survey  design  and  methods  to  meet  the  information  needs  of  the 
survey  sponsor  and  the  Army. 

c.  Collection  of  high  quality  data. 

d.  Avoidance  of  duplication  of  data  collection  efforts. 

e.  Control  of  respondent  burden. 

f.  Use  of  efficient  field  administration  procedures. 

g.  Achievement  of  high  response  rates. 

h.  Protection  of  respondent  anonymity  and  privacy. 

i.  Avoidance  of  inappropriate  topics  an  questions 
i.  Efficient  use  of  available  resources. 

2-2.  Applicability 

a.  This  chapter  pertains  to  personnel  surveys  conducted  among  U.S.  Army  Active  Duty  and 
Reserve  Component  including  the  Army  National  Guard,  Army  retirees,  and  Department  of 
the  Army  civilian  employees,  except  those  authorized  and  distributed  by  the  following: 

(1)  Office  of  the  Assistant  Secretary  of  Defense  (Force  Management  and  Personnel). 

(2)  U.S.  General  Accounting  Office. 

(3)  U.S.  Office  of  Personnel  Management  and  Merit  System  Protection  Board  (for  surveys 
conducted  of  civilian  employees  only). 

(4)  U.S.  Army  Civilian  Personnel  Evaluation  Agency. 

b.  This  chapter  also  pertains  to  surveys  of  dependent  family  members  of  Army 
Active  Duty  personnel,  when  the  purpose  is  to  determine  the  effectiveness  of  existing 
Federal  programs  related  to  Army  families  and  the  need  for  new  programs. 

c.  This  chapter  does  not  apply  to  data  collection  activities  to  obtain  factual  information. 

Factual  information  is- 

(1)  Information  defined  by  the  Code  of  Federal  Regulations,  5  CFR  1320.7(k),  such  as  facts 
and  opinions  obtained  in  connection  with  medical  examinations  and  treatment,  and 
examinations  designed  to  test  the  aptitudes,  abilities,  and  knowledge  of  persons.  Studies 
collecting  personal  data  of  this  type  should  be  approved  in  accordance  with  AR  340-21 
and  AR  70-25. 

(2)  Information  already  being  regularly  collected  and  reported  on  Government  forms,  such  as 
Standard  Forms,  Department  of  Defense  Forms,  and  Department  of  the  Army  Forms. 

2-3.  Responsibilities 

a.  The  Deputy  Chief  of  Staff  for  Personnel  (DCSPER)  will  establish  broad  policies  concerning 
personnel  survey  activities  throughout  the  Army. 

b.  The  Commanding  General,  PERSCOM,  will  exercise  overall  management  responsibility  for 
the  Army  Personnel  Survey  Program. 


40 


c  r  to 


c.  The  Director  ARI,  will: 

(1)  Establish  procedures  to  manage  the  Army  Personnel  Survey  Program,  in  accordance  with  2- 
5. 

(2)  Provide  for  the  liaison  with  and  participation  personnel  surveys  originated  by  the  Office  of 
the  Secretary  of  Defense  and  other  Department  of  Defense  (DoD)  components. 

(3)  Provide  for  the  liaison  with  and  participation  in  personnel  surveys  originated  by  the  Office 
of  the  Secretary  of  Defense  and  other  Department  of  Defense  components. 

d.  Headquarters,  Department  of  the  Army  (HQDA)  agencies  and  their  field  operating 
agencies,  Army  agencies,  MACOMs,  installations  and  their  satellite  units,  and  U.S. 

Army  elements  of  DoD  agencies  and  joint  activities  will: 

(1)  Establish  procedures  Consistent  with  this  chapter  for  authorization  and  coordination  of 
Army  personnel  surveys  conducted  only  within  the  command,  agency,  or  installation. 

(2)  Submit  survey  requests  to  ARI  for  authorization  of  Army  surveys  that  will  be  conducted 
outside  the  requesting  command,  agency,  or  installation,  in  accordance  with  paragraph  2-5. 

(3)  Ensure  that  only  authorized  personnel  surveys  are  conducted  within  the  command,  agency, 
or  installation,  in  accordance  with  paragraph  2-5. 

(4)  Place  on  hold  and  inform  ARI  of  any  survey  believed  not  authorized  in  accordance  with  this 
regulation. 

(5)  Upon  request,  provide  ARI  with  information  on  surveys  conducted  or  planned  within  the 
command,  agency,  or  installation. 

(6)  As  requested  by  the  Director,  ARI  will  assist  in  data  collection  associated  with  DA-approved 
surveys  by  providing  administrative,  logistical,  and  automated  data  processing  support. 


c. 


2-4.  Army  Sponsorship  of  Surveys 

.  A  survey  of  Army  personnel  about  Army  matters  must  have  an  Army  agency  as  a  sponsor. 

.  Sponsorship  of  a  survey  of  Army  personnel  is  reserved  for  the  activity  proponent  who  is  the 
primary  subject  of  the  survey.  Coordination  among  the  appropriate  Army  sponsors  is 
required  when  a  survey  addresses  in  depth  two  or  more  primary  subjects. 

Army  sponsors  of  non- Army-initiated  or  private  surveys  of  Army  personnel  are  responsible 
for  ensuring  that  the  requirements  of  this  chapter  are  met. 


2-5.  Authorization  Procedures 

a.  Surveys  conducted  within  a  single  command  will  be  authorized  by  the  commander,  in 
accordance  with  paragraphs  2-5.d.(l)  and  chapter  4. 

b.  Authorization  from  ARI  is  required  for  surveys  conducted  among  personnel  from  two  or 
more  Army  components,  major  commands  (MACOMs)  or  regions. 

(1)  The  Request  for  Authorization  to  Survey  Department  of  the  Army  Personnel  form  will  be 
prepared  in  accordance  with  figure  2-1  and  submitted  to  the  Commander,  ARI. 

(2)  ARI  will  conduct  a  technical  review  addressing  the  following- 

(a)  Clear  statement  of  need  for  the  information  and  how  it  will  be  used  to  benefit  the  Army 

(b)  Extent  that  the  request  meets  the  purposes  identified  in  paragraph  2-1. 

(a)  Extent  the  proposed  methods,  procedures,  and  techniques  conform  to  U.S.  Federal 
guidelines  for  statistical  activities. 

(3)  Adequacy  of  the  proposed  survey  design,  methodology,  procedures,  techniques,  the 
survey  data  collection  instrument(s),  including  questionnaire  and  survey  item  pre-testing 
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procedures,  and  any  additional  materials,  data  collection  activities,  and  analysis  plans  to 
achieve  the  purposes  of  the  survey. 

(4)  ARI  will  consult  with  the  survey  sponsor  on  the  results  of  the  review  within  15  working  days 
after  receipt  of  the  request.  Any  required  changes  will  be  identified.  Depending  on 
available  resources,  ARI  may  provide  the  survey  sponsor  with  assistance  in  making  the 
necessary  changes. 

(5)  As  the  approving  authority,  ARI  will  authorize  a  survey  by  assigning  a  survey  control 
number  (SCN)  and  requirement  control  symbol  (RCS).  For  printed  or  computer-assisted 
survey  data  collection  instruments,  the  authorizing  information  will  be  displayed  at  the  top 
of  the  cover  page,  first  page,  or  initial  screen.  For  telephone  or  personal  interview  surveys, 
the  authorizing  information  will  be  cited  by  the  interviewer  prior  to  the  interview. 

c.  Small-scale  pretests  of  draft  survey  items  and/or  data  collection  instruments  do  not  require 
authorization  if  conducted  with  fewer  than  50. 

d.  Authorization  is  not  required  for  semistructured  group  discussion  interviews  (focus 
groups).  The  group  discussion  interview  is  encouraged  because  it  is  an  effective  method 
for  identifying  the  range  of  issues  concerning  a  topic,  developing  survey  items,  reviewing 
drafts  of  data  collection  instruments,  and  addressing  other  study  purposes. 

e.  Information  releases  on  survey  results  are  controlled  by  the  survey  sponsor  and,  if  applicable, 

are  governed  by  AR  25-55  and  AR  340-21. 

2-6  Surveys  of  General  Officers  and  Senior  Executive  Service  (SES) 

a.  Usually,  general  officers  and  members  of  the  senior  executive  service  are  not  included  in  the 
samples  drawn  for  surveys  of  Army  personnel. 

b.  The  General  Officer  Management  Office  will  review  all  surveys  proposed  for  general 
officers. 

c.  The  Senior  Executive  Service  Office  will  review  all  surveys  proposed  for  SES-level  Staff 

2-7  Unauthorized  Army  Surveys 

a.  Surveys  not  approved  in  accordance  with  this  regulation  are  not  authorized  for  distribution  on 
Army  installations  or  in  Army  workplaces. 

b.  Distribution  of  unauthorized  surveys  will  be  reported  to  the  installation  commander  and  ARI. 

(1)  Responses  by  Army  personnel  to  unauthorized  surveys  will — 

(2)  Not  be  made  while  on  duty. 

(3)  Not  be  derived  from  performance  of  official  duties  or  based  on  sensitive  or  classified 
information. 

c.  Not  be  considered  as  official  responses  from  Army  personnel  or  reports  about  the  Army. 

2-8  Responses  by  individuals  to  private  surveys 

Responses  by  Army  personnel  to  private  surveys  addressed  to  them  as  private  individuals 

without  official  Army  affiliation  will  neither  be  encouraged  nor  discouraged. 


42 


Chapter  3 

Army  Survey  Programs 

3-1.  Sample  Survey  Program  (SSP) 

a.  Definition.  The  SSP  is  the  Army-wide  program  for  systematic  collection  of  information  on 

the  attitudes,  opinions,  perceptions,  behaviors,  and  characteristics  of  U.S.  Army  Active  Duty 
and  Reserve  Components,  including  the  Army  National  Guard,  Department  of  the  Army 
civilian  employees,  and  the  dependent  family  members  of  Active  Duty  personnel. 

b.  Objectives.-  the  SSP  provides- 

(1)  Resources  to  conduct  the  Sample  Survey  Of  Military  Personnel  for  the  Active  Duty 
personnel  and  the  Survey  on  Officer  Careers. 

(2)  TAS  and  support,  as  needed,  for  surveys  sponsored  by  Army  agencies. 

(3)  Services  to  conduct  special  surveys,  upon  request  and  provision  of  resources  by  the 
sponsoring  Army  agency. 

(4)  Information  about  and  from  surveys  being  conducted  of  Army  personnel. 

c.  Elements  and  procedures  for  SSP  are  described  in  chapters  4, 5,  and  6. 

3-2.  Responsibilities 

For  the  SSP— 

a.  The  DCSPER  will  establish  broad  policies  concerning  personnel  survey  activities  throughout 
the  Army. 

b.  HQDA  agencies  and  their  field  operating  agencies,  Army  agencies,  MACOMs, 
installations  and  their  satellite  units,  U.S.  Army  elements  of  DOD  agencies  and  joint 
activities,  and  the  civilian  personnel  offices  serving  these  agencies- 

(1)  Appoint  a  staff  officer  (commissioned  or  warrant)  or  DA  civilian  employee  of  comparable 
grade  to  serve  as  the  personnel  survey  control  officer  (PSCO) 

(2)  Serve  as  data  collection  activities  for  the  SSMP  and  for  special  surveys  that  require 
distribution  assistance  at  the  local  level.  The  activities  include- 

(a)  Assist  in  data  collection  for  SSP  surveys,  including  surveys  of  both  military  and  civilian 
personnel,  by  providing  administrative,  logistic,  and  automated  processing  support,  as 
required. 

(b)  Ensure  that  personnel  survey  data  collection  responsibilities  are  performed  by  personnel 
service  centers  and  their  satellites  in  accordance  with  AR  600-8-104  and  a  memorandum  of 
instruction  and  by  civilian  personnel  offices  in  accordance  with  a  memorandum  of 
instruction. 

(c)  When  requested,  complete  and  return  DA  Form  2000. 

c.  The  Command  will: 

(1)  Provide  access  to  and  products  from  the  Total  Officer  Personnel  Management  Information 
System  (TOPMIS),  Enlisted  Distribution  and  Assignment  System  (ED AS),  and  other  similar 
personnel  and  organization  files  necessary  for  planning  and  conducting  personnel  surveys 
and  responding  to  information  requests  from  Army  Staff  agencies. 

(2)  Provide  TAS  on  matters  related  to  the  information  contained  in  personnel  files  maintained  in 
data  bases. 
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Chapter  4 

Personnel  Survey  Control  Officer  (PSCO) 

4-1  Responsibilities  .  .  .  .  , 

The  PSCO  is  part  of  an  Army- wide  personnel  survey  network  responsible  for  administration  oi 

surveys  that  are  conducted  within  the  command  or  installation  for  the  Sample  Survey  Program. 


4-2  MACOMPSCO  .u' 

The  MACOM  PSCO  will  implement  command  procedures  (consistent  with  chapter  2)  tor 

authorization  of  surveys  conducted  only  within  the  command. 


4-3  Installation  or  agency  and  major  subordinate  command  PSCO. 

a.  The  installation  or  agency  PSCO  will  serve  as  the  POC  for  personnel  surveys  conducted 

throughout  the  installation  or  agency  and  satellite  units. 

b.  The  major  subordinate  command  PSCO  will  provide  assistance  to  the  installation  or  agency 

PSCO  to  ensure  efficient  management  of  surveys. 

c.  The  installation  or  agency  and  major  subordinate  command  PSCO  will- 

’  ^  Implement  command  procedures  for 

authorization  of  surveys  conducted  only  within  the  installation  or  major  subordinate 


command. 


Arrange  and  monitor,  as  needed, 


administration  of  surveys  conducted  under  the  SSP. 


survey  requirements  and  suspense  dates. 


Ensure  responsiveness  to  the  SSP 
Be  authorized  direct  contact  with  ARI, 


as  required,  to  conduct  surveys  efficiently  and  effectively. 

Provide  assistance  locally,  as 

required,  when  SSP  surveys  are  being  sent  through  distribution  directly  to  Army 


personnel.  , 

q)  Upon  request  use  command  media  and 

other  activities  to  encourage  participation  in  the  OSP  and  SSP  surveys. 

(4)  Protect  the  anonymity  of  respondents  by  ensuring  that  individual  responses  are  handled  only 
by  persons  officially  involved  in  the  administration  and  processing  of  the  survey.  Data  will  not 
be  extracted  from  the  responses  at  the  installation  or  MACOM  level  prior  to  shipment  to  ARI. 

(5)  Upon  request  by  ARI,  arrange  personnel  survey  pretests  and  response  verification  or  quality 


control  checks.  . 

(6)  Upon  request,  provide  ARI  with  materials  related  to  surveys  being  conducted  within  the 

command  or  at  the  installation. 
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Chapter  5 

Sample  Survey  Program  (SSP)  Elements  and  Procedures 
5-1  SSP  Mission 

In  support  of  the  Army  Personnel  Survey  Program,  the  SSP  has  the  following  mission: 

a.  Authorize  and  coordinate  surveys  conducted  in  two  or  more  MACOMs  (Chapter  2). 

b.  Conduct  the  following  surveys: 

(1) .  Sample  Survey  of  Military  Personnel 

(2) .  Survey  on  Officer  Careers 

(3) .  Special  Surveys 

c.  Provide  TAS  on  conducting  surveys  and  studies  and  provide  information  on  surveys 

5-2.  Sample  Surveys 

a.  ARI  conducts  Army- wide,  omnibus  sample  surveys  to  accommodate  personnel  proponents 
and  other  Army  agencies  whose  information  needs  do  not  require  or  justify  separate  survey 

efforts. 

b.  Use  of  omnibus  sample  surveys  reduces  the  number  of  surveys  as  well  as  the  respondent 
burden  for  Army  personnel. 

c.  ARI  will — 

(1)  Prepare  the  data  file 

(2)  Conduct  analyses  in  accordance  with  a  basic  plan  developed  with  the  proponents  of  the 
items 

(3)  Prepare  data  tables 

(4)  Analyze  the  data 

(5)  Write  a  report  of  the  findings  containing  components  standard  in  the  survey  research 
industry. 

5-3.  Technical  Advisory  Service  (TAS) 

a.  ARI  provides  TAS  in  survey  methodology  and  procedures  to  Army  sponsors  of  personnel 

surveys.  TAS  also  may  be  provided  to  other  Army  and  DOD  components  planning  to 
conduct  surveys  or  other  studies  of  Army  personnel. 

b.  TAS  may  be  provided  for  survey  methods  and  procedures  in  the  areas  such  as- 

(1)  Survey  design. 

(2)  Sampling. 

(3)  Item  construction. 

(4)  Survey  data  collection  instrument  construction. 

(5)  Pretesting. 

(6)  Data  collection  methods  and  procedures. 

(7)  Survey  administration. 

(8)  Data  reduction. 

(9)  Data  analysis. 

(10)  Statistical  report  writing. 

c.  Sponsors  of  surveys  of  Army  personnel  are  encouraged  to  establish  working  relationships 

early  with  ARI  to  ensure  that  requests  for  survey  coordination  and  authorization  meet  all 
requirements  of  both  the  sponsor  and  ARI. 
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5-4  Data  File  Documentation  and  Archive 

a  The  purpose  of  data  file  documentation  and  the  archiving  of  research  data  is  to  preserve 
existing  research  data  files,  in  a  central  location,  that  may  prove  useful  for  future  research. 

b.  Data  files  will  be  documented  and  archived  by  the  Army  Research  Institute. 

c.  Archived  data  is  for  internal  use  only  and  will  be  used  to  support  research  activities. 

5-5.  Release  of  Information  ■  .  J  ,  .  .  . 

a.  ARI  provides  information  about  surveys,  which  have  been  conducted,  are  being  conducted,  o 

are  planned. 

b.  Survey  data  and  reports  will  be  released  by  the  survey  sponsor  and,  if  applicable,  are 
governed  by  AR  25-55  and  AR  340-211 
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Chapter  6 
Special  Surveys 

6.1.  Requests  to  conduct  special  surveys 

Upon  formal  request  and  provision  of  required  resources,  ARI  will  conduct  a  separate,  special 
survey  or  provide  the  TAS  necessary  for  conducting  a  survey  of  Army  personnel  (see  the  listing 
of  TAS  areas  in  para  6-3.b.). 
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GLOSSARY 


Section  I 
Abbreviations 

ADP 

Automatic  data  processing 

CFR 

U.S.  Code  of  Federal  Regulations 

DA 

Department  of  the  Army 

DCSPER 

Deputy  Chief  of  Staff  for  Personnel 

DOD 

Department  of  Defense 

FY 

Fiscal  year 

HQDA 

Headquarters,  Department  of  the  Army 
MACOM 

Major  Army  command 

OCONUS 

Outside  continental  United  States 

ODCSPER 

Office,  Deputy  Chief  of  Staff  for  Personnel 

OSD 

Office  of  the  Secretary  of  Defense 

OSP 

Occupational  Survey  Program 
PSC 

Personnel  service  center 

PSCO 

Personnel  survey  control  officer 


RCS 

Requirement  control  symbol 


SCN 

Survey  control  number 

SSMP 

Sample  Survey  of  Military  Personnel 

SSP 

Sample  Survey  Program 

TAS 

Technical  advisory  service 
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Section  II 
Terms 


A  DA  form  for  recording  survey  responses.  The  Army  Survey  Personnel  Program  usually  uses 
answer  sheets  or  forms,  which  are  optically  scanned  to  produce  the  database  for  a  survey. 


ForthePpumoses  of  this  regulation,  "Army  personnel"  includes  soldiers  serving  on  active  duty  with 
the  Army,  members  of  the  Army  Reserve,  retired  members  of  the  Army,  and  DA  civilian 
employees.  (Dependent  family  members  of  the  above-referenced  personnel  are  mcluded  as  Army 
personnel  in  accordance  with  Section  804,  Public  Law  100-145.)  . 


Attitude  and  opinion  survey  .  _  , 

A  formal  survey  on  the  attitudes,  opinions,  behaviors,  and  knowledge  of  people  concerning  gene  a 
and  specific  issues,  activities,  and  objects.  The  term  may  be  used  to  indicate  that  the  Army 
personnel  survey  is  not  an  occupational  survey. 


Attitudes  and  opinions  .  ... 

An  attitude  is  a  general  orientation  to  a  patterned  way  of  thinking  about  something 

leads  to  an  opinion  on  a  specific  aspect  of  something. 


An  attitude 
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A^urvev  that  has  been  assigned  an  SCN  in  accordance  with  this  regulation  or  by  DOD  or  the  U.S. 
Office  of  Management  and  Budget.  A  survey  conducted  in  two  or  more  MACOMs  must  have  an 

SCN  assigned  by  ARI. 

Behaviors 

Activities  or  actions  performed  by  people. 

A^elephone^tervfew  survey  which  uses  a  computer  software  program  to  assist  the  interviewer  in 
recording  responses  and  asking  questions  in  an  appropriate  sequence. 

Data  analysis 

Use  of  statistical  procedures  for  reporting  aggregate  responses  to  a  survey. 

responsible  for  collecting  job-related  information  from  soldiers  withm  their 
areas  of  jurisdiction  whom  are  being  surveyed  under  OSP  or  SSP. 

Pro^edm^foTpreparing  for  statistical  analyses  the  responses  to  questions  from  a  survey  data 
collection  instrument. 

wSJrn  DA  Form  3408  (OSP  Data  Sheet),  containing  TOE,  TDA,  ASI  and  authorization 
information  pertaining  to  an  enlisted  MOS  job  incumbent. 

The  Squarterly^sthig  *  of  surveys  being  conducted  among  Army  personnel  and  sponsored  at  the 
MACOM  or  higher  Army  or  DOD  level. 

UsTof  AeU-Tpostal  Service,  small  package  shipper,  and/or  U.S.  military  mail  as  a  means  for 
distributing  data  collection  instruments  and  other  materials  directly  to  respondents  and  for  the 
direct  return  of  completed  materials  to  a  central  point. 

Completion  of  a  self-administered  data  collection  instrument  in  a  group  setting,  with  a  ,SUI^ey 
administrator  providing  instructions  on  how  to  complete  the  data  collection  instrument  and  other 
information  related  to  the  survey.  Completed  data  collection  instruments  are  returned  to  the  survey 

administrator. 


Any  smtement  of  fact  or  opinion,  whether  in  numerical,  graphic,  or  narrative  form,  and  whether  in 
oral  or  maintained  on  paper,  magnetic  tapes,  or  other  media  [5  CFR  1320.7(k)J. 
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Nfflia*iwti™tement  describing  the  sponsor,  purposes,  and  procedures  planned  for  conductmga 
survey  of  Army  personnel  and  the  dependent  family  members  of  Active  Duty  soldiers.  For  Army 
personnel  survejs  conducted  of  personnel  in  two  or  more  MACOMs  the ^  request  must  te 
submitted  to  USAPIC.  Some  surveys  also  may  have  to  be  approved  by  the  U.S.  Office  oi 
Management  and  Budget. 

Surveys  conducted  among  Army  personnel  by  persons  or  organizations  outside  of  the  Army. 
Non- Army  surveys  must  have  an  official  Army  sponsor  in  order  to  be  considered  for  authorization 

and  distribution. 

A  survey  of  personnel  on  various  aspects  of  their  work  to  include  tasks  performed,  and  the  skills, 
knowledge  and  abilities  necessary  to  do  the  work.  This  includes  surveys  that  pertain  to  collection 
of  various  aspects  of  job  training  requirements  (task  criticality)  as  determined  by  supervisory  and 
management  personnel.  Other  unique  factors  may  be  collected  in  order  to  meet  umque  analysis 

requirements. 

Personnel  Survey  Control  Officer  (PSCO)  ,  . 

The  appointed  individual  at  a  MACOM,  command,  installation,  or  major  satellite  command  who  is 
responsible  for  receiving,  distributing,  accounting  for,  and  returning  Army  personnel  surveys. 

Personal  interview  .. 

Completions  of  a  data  collection  instrument,  using  an  interviewer  to  ask  questions  and  record  the 

answers  of  the  respondent.  It  may  be  done  face-to-facc  at  the  same  location  or  by  telephone. 

Personnel  survey 

The  collection  of  information  from  or  about  any  population  or  group  for  the  purpose  of  studying 
characteristics  of  the  population  or  group. 

The  specific  group  or  type  of  people  selected  to  participate  in  a  survey.  A  population  may 
comprise  several  specific  subpopulations. 

Pretest/Dilot  test 

A  test  of  survey  procedures  and  a  draft  of  items  or  the  data  collection  instrument  to  determine  their 
appropriateness.  A  pretest  is  conducted  with  a  small  number  of  people  (usually  from  10  to  50). 
Revisions  of  the  planned  procedures  and  data  coUection  instrument  are  based  on  the  findings  ol  the 
pretest.  Several  pretests  may  be  required. 


Proponent 

The  official  Army  sponsor  of  an  Army  activity. 
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A^alection  instrument  used  by  an  interviewer  and  respondent  or  by  a  respondent  alone 
during  a  personnel  survey. 

Irfsurveyresearch,  the  likelihood  that  a  question  will  be  answered  with  the  same  response  if  asked 
again  a  short  time  later. 

A^ymboTuseVby  the  Army  to  indicate  that  a  specific  data  collection  activity  has  been  authorized. 

Sn  of  a  larger  group  of  people.  The  portion  is  selected  by  some 
so  that  each  person  in  the  larger  group  has  a  known  probability  of  being  ■  selected.  Whe 
probability  of  being  selected  is  known,  inferences  can  be  made  to  the  larger  group  based  on  t  e 
findings  for  the  smaller  group. 

Sample  design/sampling  plan  _ 

Procedures  established  for  selecting  the  population  and  subpopulations  for  a  survey.  The 
procedures  take  into  account  the  number  of  persons  in  the  total  population  being  examined,  the 
desired  sampling  error  and  confidence  intervals  required,  and  the  expected  response  rates  for  each 

subpopulation. 

Sample  Survey  of  Military  Personnel  (SSMP)  nQAPTr 

The  general-purpose,  omnibus  survey  of  Active  Duty  Army  personnel  conducted  by  US  ^ 
semiannually:  It  has  been  conducted  regularly  since  it  was  first  conducted  in  December  1943  as 
the  Personnel  Survey  of  the  Army. 

The  Subset  of  a  total  population  being  studied.  For  example,  officers,  enlisted  personnel, 
dependent  family  members  of  soldiers,  and  DA  civilians  are  four  basic  subpopulations  of  the  total 

Army  population. 

Ageneral  term  used  to  describe  the  process  of  obtaining  information  systematically  from  a  selected 
group  of  people  through  the  use  of  a  data  collection  instrument  with  similar  questions  and  response 
categories.  A  survey  asks  identical  question(s)  of  10  or  more  persons. 


54 


^generaJ^rm^use^to  describe  how  a  data  coUection  instrument  will  be  distributed  to  and 
completed  by  respondents  and  how  the  responses  will  be  collected  at  a  central  processing  point. 

Survey  control  number  (SCN)  „  .  . 

Number  assigned  by  USAPIC  and  placed  on  the  front  cover  of  a  data  collection  instrument  to 

indicate  that  the  survey  has  been  authorized  distribution  by  the  Army. 

Process  bTwhichttie  Army  identifies  surveys  being  conducted  at  the  MACOM  or  higher  level  and 
uses  the  information  to  ensure  effective  and  efficient  use  of  soldier  time  and  Army  resources. 

Procedures* planned  for  conducting  all  aspects  of  a  survey,  including,  but  not  limited  to, 
identification  of  purposes  and  specific  information  needs,  sample  design,  data  collection 
instrument  construction,  survey  administration,  data  reduction,  and  data  analysis. 

Acoordhiated^stematic  collection  of  information  by  surveys  of  Army  personnel  to  address  the 
needs  of  personnel  proponents  and  other  agencies  within  the  Army.  It  includes  identification  of 
information  needs;  survey  design;  survey  administration;  data  reduction,  processing,  and  analysis, 
and  reporting  the  results. 

Completion  of  a  data  collection  instrument  through  the  use  of  a  telephone  to  link  the  respondent 
with  the  interviewer.  The  interviewer  records  the  answers  provided  by  the  respondents. 


^survey  research,  the  likelihood  that  the  responses  obtained  are  an  accurate  reflection  of  what 
actually  exists  in  terms  of  the  attitudes,  opinions,  behaviors,  and  knowledge  being  studied. 
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Figure  2-1 

REQUEST  FOR  AUTHORIZATION  TO  SURVEY 
DEPARTMENT  OF  THE  ARMY  PERSONNEL 

THE  ARMY  SAMPLE  SURVEY  PROGRAM 
(AR  600-XX) 

TO:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
ATTN:  TAPC-ARI-PS 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 

Tel:  DSN  667-7801/7802 

Commercial  (703)  617-7801/7802 

Date  Submitted: 

(Instructions:  Provide  the  following  information.) 

1.  Name  of  sponsoring  Army  or  DOD  organization  or  office. 

2.  Point  of  contact. 

3.  Mailing  address. 

4.  Telephone  number,  AUTOVON  and  commercial. 

5.  Recommended  title  of  the  survey. 

6.  Proposed  schedule. 

a.  Completion  of  draft  survey  data  collection  instrument  date. 

b.  Survey  pretest  dates. 

c.  Survey  administration  dates. 

d.  Data  analysis  dates. 

e.  Final  report/briefing  dates. 

7.  Justification  of  survey  request. 

a.  Describe  the  general  purpose  of  the  survey. 

b.  List  the  specific  objectives  being  addressed  by  the  survey. 

c.  Describe  how  the  survey  results  will  be  used. 

8.  Background  research. 

a.  Describe  the  planning  and  coordination  of  the  survey,  with  a  focus  on  Army  organizations/oftices 
contacted. 

b.  Describe  the  most  recent  relevant  research.  Identify  key  publications,  articles,  and/or  papers  reviewed. 
Include  both  military  and  civilian  sources 

9.  Target  population 

a.  Describe  the  specific  groups  or  subgroups  of  Army  personnel  to  be  surveyed. 

b.  Describe  the  reasons  why  the  specific  groups  or  subgroups  have  been  selected.  , 

10.  Suggested  sample  size. 

a.  Estimate  the  total  size  of  each  group  or  subgroup. 
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b.  Estimate  how  many  persons  in  each  group  or  subgroup  will  be  asked  to  participate  m  the 
survey. 

1 L  T Wenttfy^pretod  method  for  data  collection  (how  the  data  collection  instrument  will  be  distributed  ai 
b  Estimate  the  average  time  required  by  a  respondent  to  complete  the  survey.  . 

12.  Data  collection  instrument  and  other  survey  materials.  (Attach  survey  instrument,  memorandums/letters,  an 

13  ^Describe  how  the  data  will  be  processed  (hand-tabulated,  PC  software  program,  optically  scanned,  etc.), 
b.  Describe  how  the  data  will  be  analyzed  (frequencies/percentages,  cross-tabulations,  or  other  statistical  prc 

14.  Release  of  survey  results  by  ARI.  (Select  one.) 

a.  Release  to  any  Army  requester/media 

b. Release  to  only  that  requesters/media  specified.  (Specify.) 

c.  Release  is  not  desired.  (Give  reasons.) 
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Figure  2-2 

REQUEST  FOR  SURVEY  ITEM  APPROVAL 
FOR  SAMPLE  SURVEY 

THE  ARMY  SAMPLE  SURVEY  PROGRAM 
(AR600-XX) 

TO:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
ATTN:  TAPC-ARI-PS 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 

Tel:  DSN  667-7801/7802 

Commercial  (703)  617-7801/7802 

Date  Submitted: 

(Instructions:  Provide  the  following  information.) 

1.  Name  of  sponsoring  Army  or  DOD  organization  or  office. 

2.  Point  of  contact. 

3.  Mailing  address 

4.  Telephone  number,  AUTOVON  and  commercial. 

5.  General  topic  addressed  by  requested  items. 

6.  Justification  of  survey  request. 

a.  Describe  the  general  purpose  of  the  survey  items. 

b.  List  the  specific  objectives  being  addressed  by  the  items. 

c.  Describe  how  the  results  will  be  used. 

7.  Background  research. 

a. Describe  the  planning  and  coordination  of  survey  items,  with  a  focus  on  Army 
organizations/offices  contacted. 

b. Describe  the  most  recent  relevant  research.  Identify  key  publications,  articles,  and/or  papers 
reviewed.  Include  both  military  and  civilian  sources. 

8.  Requested  items.  (Attach  draft  of  proposed  survey  items.) 

9.  Special  data  analysis  requirements. 

a.Describe  how  the  data  should  be  analyzed  in  addition  to  reporting  the  results  by  paygrade 
groupings  (such  as  by  sex,  marital  status,  CONUS/OCONUS  location), 
vindicate  whether  assistance  in  interpreting  the  results  will  be  needed. 

10.  Release  of  survey  results  by  ARI.  (Select  one.) 

a.  Release  to  any  Army  requester/media. 

b.  Release  to  only  that  requesters/media  specified.  (Specify.) 

c.  Release  is  not  desired.  (Give  reasons.) 
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